[10191/1620] 

METHOD OF CALIBRATING A SENSOR SYSTEM 
Fi eld Of The Invention 

The present invention relates to a method of calibrating a sensor system, in particular an 
image processing system. 

Background Information 

It is known that image recording or camera systems and other sensor systems are used as a 
module of a vehicle safety system in road traffic. In this case, information regarding the 
distance and relative velocity of the vehicle is constantly being processed in relation to other 
objects, i.e., other vehicles, and the prevailing road conditions. The image recording systems 
and optionally also radar sensors are used to measure geometric quantities in the vehicle 
environment; e.g., radar sensors are known from German Published Patent Application No. 
42 42 700 Al, for example. With this known arrangement, a cruise control, for example, in a 
vehicle can be expanded so that driving speed is adapted to slower vehicles in front if the 
sensor systems detect such vehicles in the anticipated path of the vehicle. For example, the 
path of the vehicle can be determined with the help of yaw sensors, steering angle sensors or 
transverse acceleration sensors, on the basis of wheel speeds or also with the above- 
mentioned image processing systems or navigation systems. 

For satisfactory functioning of the system, prior calibration of the sensor system or the 
individual sensors in the vehicle environment is done in principle for a subsequent accurate 
measurement of geometric quantities such as lengths, speeds, etc.; this is done in a 
laboratory-like environment either before or after installation of the sensors in the vehicle. 
The various calibration methods usually require controlled movement of the sensors or the 
objects detected by the sensor systems. Often it is even desirable to observe a specific 
artificial object, which is then referred to as a calibration field accordingly. To guarantee 
permanent functional reliability, subsequent repeated checking of the calibration for possible 
changes is desirable, which may be very complicated. 
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It is known from European Patent No. 602 013 that ambient information available from a 
road map, for example, is used to calibrate a compound navigation system. In this case, 
information regarding the driver's own vehicle status is processed in combination with known 
information about the environment available from the map. 

5 

Summary Of The Invention 

A method of calibrating a sensor system which is used to detect and analyze objects in the 
path of a vehicle is carried out according to the present invention in such a way that 
characteristic data on the objects, including the roadway, is detected with the sensor system in 

10 an advantageous manner, and data recognized as stationary or quasi-stationary objects, taking 
into account the movement of the vehicle, is sent to a calibration unit. The deviation in the 

£" data measured instantaneously from data provided by a model of the objects is determined as 

(: an error vector in this calibration unit and used to correct the data of the model for the 

J: purpose of minimizing the deviation, e.g., by an iterative method. 

3 5 

An automatic calibration of a sensor system can be performed in an especially advantageous 
manner according to the present invention, which also permits subsequent automatic checking 
-[ of a calibration that has already been performed. To do so, no other equipment is necessary 

=- except for the essentially known sensor elements in the vehicle environment and an analyzer 

™20 unit or calibration unit. 

Furthermore, no special movements of the vehicle which previously had to be executed 
separately for the purpose of calibration are necessary here because the calibration according 
to the present invention instead utilizes the vehicle's own movement in operation as well as 

25 the knowledge that objects in the three-dimensional world are often rigid and that they move 
as a whole, and the calibration of the sensors is stable over a longer period of time. In this 
case, the calibration of the sensor system is quasi-permanent, i.e., it is variable but only 
gradually. Heuristic assumptions regarding the typical behavior of other vehicles (e.g., 
remaining in their lane) which can result in malfunctions in practical applications are not 

30 necessary for this method. Thus, one property of the present invention is that it permits a 

calibration which detects the measurable quantities as well as possible. The demand for a so- 
called true calibration, such as that in the case of a laboratory calibration, is only secondary 
here. 
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In addition, from the standpoint of manufacturing technology, installation of uncalibrated 
sensors is especially advantageous, because the calibration of a sensor is often sensitive to 
mechanical or thermal stress. Thus, not only is manufacturing simplified, but also it is readily 
possible to later take into account the effects of the installation or any other changes in the 
sensor system or the vehicle on the calibration. In addition, the calibration can be checked at 
any time. 

According to an especially advantageous embodiment of the method according to the present 
invention, there is a first determination of object data which is stored as model data in an 
initialization phase with preselectable parameters. In all subsequent measurements which are 
performed on a cyclic basis, the respective instantaneous object data is processed in the 
calibration unit with the previously determined and stored model data to obtain the respective 
error vector. 

During the processing of data in the calibration unit, the recurring object data from previous 
measurements is selected, deleting object data not found again and including object data 
newly added. The object data showing a reduction in the respective confidence interval after 
repeated measurements from different positions of the vehicle is characterized as data 
belonging to a stationary or quasi-stationary object. 

In the case of the method according to the present invention, a corresponding relative velocity 
of objects can also be determined from successive object data and then used to determine the 
vehicle's own movement. Object data attributable to an object having this same relative 
velocity can be characterized as data belonging to a stationary or quasi-stationary object. A 
rotational motion of the vehicle due to pitching and/or cornering can also be used as the 
vehicle's own motion in an advantageous manner. 

The results of calibration of one sensor of the sensor system can be easily transferred to one 
or more other sensors on the motor vehicle for calibration of these sensors as well. Thus, with 
the method according to the present invention, joint calibration of most of the vehicle sensors 
after installation is possible without requiring any special boundary conditions. In particular, 
the combination of sensors such as camera, radar, wheel sensor, acceleration sensor is 
advantageous here, but an application in conjunction with LIDAR sensors or ultrasonic 
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sensors is also possible. 



For the case when the sensor(s) of the sensor system yield(s) contradictory measurement data, 
a signal may be transmitted to an analyzer unit or to the driver of the vehicle, for example. 

5 

The present invention can thus be used to advantage to perform a calibration with good 
results without requiring any special calibration arrangement or a calibration motion 
sequence. The sensor system or analyzer system may be completely or largely automatically 
self-calibrating and it also may determine the accuracy and reliability of its own calibration. 
10 Parameters that are not or have not yet been calibrated are characterized by an infinite 
~ inaccuracy. In particular, the method according to the present invention notices major 

changes in calibration which are possible due to thermal or mechanical shifts, for example. 

~- The present invention can be used to particular advantage when the sensor system has an 
4 5 image recording system where pixels are detected and analyzed serially by an electronic 
camera having a nonlinear transformer characteristic in the recording intervals. Many 
traditional image recording methods perform the camera calibration by the indirect route of 
fundamental matrices and therefore rely on simultaneous recording of the image elements 
(cubic image grid) and synchronous sensor data. The method according to the present 
20 invention, however, also operates with the individual pixels recorded at any desired point in 
time and with asynchronous sensor data. Thus, the method according to the present invention 
permits in particular calibration of high-resolution video cameras having a nonlinear 
characteristic which usually record their pixels serially. 

25 Brief Description Of The Drawings 

Figure 1 shows a diagram of a vehicle having a sensor and an analysis and calibration system 
for use with a plurality of individual sensors including an image recording system. 

Figure 2 shows a first flow chart of the calibration steps. 

30 

Figure 3 shows a second flow chart of the calibration steps. 

Figure 4 shows a diagram of a pinhole camera model to illustrate the theoretical relationships 
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in obtaining an error vector in the calibration method. 
Detailed Description 

Figure 1 shows a possible embodiment of a sensor system for carrying out self-calibration of 
5 the sensor elements; this sensor system is integrated into a system for detecting the vehicle 
environment in a vehicle 1, as mentioned in the introduction to the description. Figure 1 
shows an electronic camera as an image recording sensor 2, a radar sensor 3, an acceleration 
sensor 4 and additional wheel sensors 5, these sensors being used only as an example. These 
sensors 2 through 5 may be oriented for detecting the motion of vehicle 1 and the 

10 environment in front of the vehicle, as indicated in Figure 1 with the radiating fields and the 

-j- arrows for the mechanical quantities to be detected. 

In the case of given factors typical of the application, there are objects 6 or even groups of 
objects 6 in the detection range or in the vehicle environment, which are not deformed over a 

4 5 long period of observation but instead move as a whole. For example, the roadway, crash 

barriers or under some circumstances even other vehicles may be mentioned in this regard in 
particular. In the embodiment according to Figure 1, the data obtained with sensors 2 through 
5 is sent to one or more analyzer units 7 which also contain a calibration unit and analyze the 
Z- information about the vehicle environment and perform the calibration. Analyzer unit 7 is 

20 usually present anyway to determine vehicle environment information and it can therefore 
also be used to determine the calibration parameters. 

Analyzer unit 7 requires object data which can be obtained from a memory 8 in order to 
perform the sensor calibration. With the analysis of the object data, the calibration 

25 information in memory 8 is improved in analyzer unit 7 with the analysis of the object data in 
a manner to be explained below. The information about the vehicle environment as calculated 
by analyzer unit 7 with the help of the calibration is then converted by an actuator system 9 to 
influence vehicle 1 and/or is relayed as information to the driver of vehicle 1 over a man- 
machine interface 10 such as a loudspeaker or a display. Actuator system 9 may process a 

30 control signal for the vehicle brakes, for example. 

Possible process steps in carrying out the self-calibration of the sensor system in vehicle 1 
according to Figure 1 are illustrated in Figure 2, where the quantities of a real world, 
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represented by blocks 21 and 22 and measured by sensor elements 2 through 5 are compared 
with modeled quantities, represented by blocks 22 and 23. The difference between the 
measured quantities and the modeled quantities is determined in block 24 and forms an error 
vector in block 25. The calibration is performed within analyzer unit 7 in such a way that the 
5 error vector is as small as possible; in other words, the model world corresponds to the real 
world as closely as possible. 

The calibration data obtained from the object data of sensors 2 through 5 together with the 
model data of the parametric model world, corresponding to the description of the vehicle 
environment also form a parameter vector which is shown abstractly in a block 26. In 
addition to the parametric description of the world, the model here also contains the world 
mapped on quantities that can be measured by sensor elements 2 through 5. For the example 
of the use of a camera as an image recording sensor 2, this is the projection of the world onto 
the image sequence. The measurable quantities are then determined from the parametric 
model world as well as being determined by sensor element 2. The difference between these 
two determinations of the measurable quantities is grouped to form an error vector (block 25). 
The parameter vector in block 26 is then adjusted through suitable methods which are 
characterized in a block 27 so that the error vector is minimal, i.e., it is minimized so that the 
actual measurements and the quantities determined by the model in conjunction with the 
instantaneous parameter vector correspond as well as possible. 

A simple criterion for a good correspondence of the quantities for carrying out such a 
minimization method which is essentially known would be, for example, the sum of squares 
of the deviations. A sufficiently good result can be obtained in the minimization method by 
25 way of a so-called least square method, or it is also possible to use other solid calculation 
methods as well as an extended Kalman filter or a similar approach. All of these methods 
which are known per se make it possible to determine the accuracy and reliability of 
parameters and measurements. 

30 The method according to the present invention is based on the assumption, as mentioned 
above, that rigid objects 6 or groups of such objects 6 exist in the vehicle environment. 
Therefore, their motion can be described completely by three rotational parameters and three 
translational parameters. In addition, it is assumed that at least some of these objects 6 are 
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moving relative to vehicle 1, e.g., on the basis of the characteristic motion of vehicle 1. These 
prerequisites are often satisfied to a sufficient extent so that it is possible to calculate or check 
the calibration in short intervals. 

Figure 3 shows another possible flow chart for carrying out the method according to the 
present invention with regard to the analysis of the object data which is explained in 
conjunction with a so-called pinhole camera model on the basis of Figure 4. A simple method 
of analyzing object data is explained below on the basis of the flow chart illustrated in Figure 
3. The basic sensor system includes only one camera (sensor 2) and radar sensor 3. A detailed 
explanation of the flow chart according to Figure 3 will be given after a description of Figure 
4. 

Figure 4 shows as an example a simple known mapping model for sensor 2 as a pinhole 
camera model and is used only for a definition of terms. The calibration parameters of sensors 
2 and 3 are combined in a parameter vector together with model quantities of the parametric 
model world as a description of the vehicle environment as explained with reference to Figure 

2. The model according to Figure 4 also contains, in addition to the parametric description of 
the world, the mapping of the world on the quantities that can be measured with sensors 2 and 

3. For the example of camera 2 or the pinhole camera according to Figure 4, this is the 
projection of the world onto the image sequence. The measurable quantities are then 
determined from the parametric model world as well as measured by the sensor system. The 
difference between these two determinations of the measurable quantities is grouped to form 
an error vector. The parameter vector is adjusted through suitable methods so that the error 
vector is minimal, i.e., so that the actual measurements and the quantities determined by the 
model in conjunction with the instantaneous parameter vector correspond as closely as 
possible. In Figure 4, point C describes the projection center through which all the imaging 
rays of sensor 2 pass. The perpendicular to the plane of the image through projection center C 
is referred to as the optical axis (Z axis). The Z axis intersects an image plane B at main 
image point P BH . Main image point P BH has the coordinates (x_r, y_r) corresponding to (x h, 
y_h) in the computer coordinate system spanned by image lines and image columns. The X 
and Y axes of the camera coordinate system are parallel to the axes of the computer 
coordinate system. The distance of the projection center from the main image point P BH 
divided by the distance between two pixels of an image column is referred to as camera 
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constant c. Finally, the ratio of the distance between two pixels within an image column to the 
distance within one image line is referred to as axis ratio s. For the pinhole camera model 
according to Figure 4, the four parameters mentioned here (x_h, y_h, c, s) form the intrinsic 
parameters of calibration. Thus, the projection of an object point P 0 onto the image plane 
5 (pixel P B ) can be given by the following equation: 

x_r = x_h + s*c*X / Z 

y_r = y_h + c*Y / Z (1) 

10 It is also assumed here that a simple pinhole camera model according to Figure 4 is used as 
the basis for the camera (sensor 2), its intrinsic calibration including the parameters: main 
image point P BH , camera constant c and image axis scaling. In addition, shift vector T and the 
rotation R between the radar coordinate system and the camera coordinate system should be 
calibrated for radar sensor 3. This yields the relationship between the camera coordinate 

15 system (X, Y, Z) and the radar coordinate system (X_R, Y_R, Z_R) as follows: 

(X,Y,Z) = R * ((X_R,Y_R,Z_R) - T) (2) 

The calibration parameters of sensors 2 and 3 then form the calibration vector p_k which is 
20 sought. 

The parameters that can be measured by the camera (sensor 2) include, for example, the 
image coordinates of stationary characteristic points on objects 6 (corners, etc.), and those 
that can be measured by radar sensor 3 include the distance, angle and relative velocity of 

25 stationary objects 6 in the radar coordinate system. The geometric parameters from which 
measurable quantities can be determined with the help of the calibration on the basis of the 
imaging model according to Figure 4 are referred to here as model parameters p_m. For the 
selected example, these are the (camera) coordinates of the characteristic points and the 
coordinates and the relative velocity of objects. The imaging model is given by equation (1) 

30 for the camera (sensor 2) and by equation (2) for radar sensor 3. Model parameters pjm 
together with calibration parameters p_k form the parameter vector p (see block 26 from 
Figure 2). 
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According to the flow chart in Figure 3, parameter vector p is first initialized (block 30), e.g., 
the main image point P BH is arbitrarily set at the midpoint of the image, and approximate 
values such as those given by the camera manufacturer are used as camera constant c and the 
axis ratio. Shift vector T and rotation vector R are measured approximately, e.g., with a tape 
5 measure. Since neither characteristic pixels nor objects have been measured so far, the model 
parameter vector p_m still does not contain any parameters and the set of measurable 
quantities is initialized as empty. 

In a subsequent step, measurements are performed by sensors 2 and 3 (block 31) with the 
10 measurable quantities already determined in previous measurements being sought again in 

particular in block 32; this is called tracking. The instantaneous set of measurable quantities 

is then expanded after block 33 by adding new measurable quantities such as new 
(-' characteristic points that have recently entered the image. The measurable quantities not 

found again in tracking are then removed from the set of measurable quantities. 

15 

For all measurable quantities, the model parameters p_m and the vehicle's own motion at the 
: J previous measurement time are determined with the respective confidence intervals by way of 
■f known standard methods. After a characteristic object point P Q has been measured the first 

f time, for example, the confidence interval still includes at least one straight line. After 
20 repeated measurements of the same characteristic object point P 0 from different positions, the 
model parameters p_m and the vehicle's own motion can be measured with progressively 
greater accuracy, so that a corresponding confidence interval becomes smaller. Object points 
P G with a small confidence interval and coordinates that are stable in time are characterized as 
belonging to stationary objects 6 and are included in the model parameter vector. Conversely, 
25 points that are not stationary or are no longer stationary because of variable coordinates are 

removed from the model parameter vector. Similarly, the vehicle's own motion is determined 
by way of solid estimation methods from the relative velocities of all objects 6 as measured 
by radar sensor 3, and only the objects that move at this relative velocity are accepted as 
stationary and their position assigned to the model parameter vector. 

30 

In contrast with methods using what is know as active observation, thus no special motion of 
vehicle 1 is necessary for the method according to the present invention as described on the 
basis of the embodiment, but instead the existing motion of vehicle 1 is determined and 
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utilized. With the help of mapping equations (1) and (2) formulated above, the measurable 
quantities from the model world are then determined from the parameter vector p according 
to block 34 from Figure 3. The difference between the value of the measurable quantities on 
the basis of the sensor measurement and the modeling is known as the error vector (block 35) 
5 with the error vector also depending on the calibration parameters because the modeled 
values depend on the calibration. 

Due to an adjustment calculation as mentioned above, parameter p can be varied according to 
block 36 so that the error vector becomes smaller, e.g., by a least squares method. Then new 
10 measurements by sensors 2 and 3 can be added so that on the whole this yields an iterative 
calibration method (block 37). In performing the process steps described here, a relatively 
rapid convergence of the calibration parameters to good values for geometric measurements. 
The use of rotational motions of vehicle 1 such as those which occur in practice due to 
pitching or turning a corner is especially favorable for the quality of the calibration. 
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